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1.0  INTRODUCTION 


This  Final  Report  describes  the  results  of  work  performed 
by  Decisions  and  Designs,  Inc.  (DDI)  under  DARPA  contract  num¬ 
ber  MDA903-80-C-0194.  The  goal  of  this  contract  was  to  develop 
a  prototype  version  of  a  user-friendly  decision  aid  which  could 
be  used  by  persons  with  little  or  no  prior  training.  A  proto¬ 
type  system  has  been  developed,  and  a  working  model  has  been 
delivered  to  DARPA;  this  report  is  intended  to  accompany  that 
working  model.  Its  function  is  to  document  the  purpose  of  the 
research  effort,  the  technical  approach,  accomplishments  of  the 
proiect,  and  the  prognosis  for  future  development  based  on  lessons 
learned  during  the  effort.  Because  of  the  aid's  self-explanatory 
nature  and  its  unique  hardware  configuration,  neiriiei  a  users 
guide  nor  a  system  manual  is  needed.  Furthermore,  no  attempt 
is  made  to  describe  the  details  of  the  aid's  functions  and  dis¬ 
plays,  except  insofar  as  that  information  reflects  higher-order 
technical  issues. 

1 . 1  Objectives 

The  motivation  for  research  and  development  of  a  Friendly 
Decision  Aid  stems  from  two  historical  trends.  On  the  one  hand, 
the  mathematical  techniques  of  Decision  Analysis  have,  during 
the  past  decade,  achieved  a  high  degree  of  technical  sophistica¬ 
tion;  this  growth  reflects  not  only  the  success  of  a  long-term 
research  effort  sponsored  by  DARPA  and  several  other  government 
agencies,  bur  also  the  practical  experience  gained  through  myriad 
applications  by  decision-analytic  firms  such  as  DDI.  At  the  same 
time,  the  very  success  of  the  decision-analytic  techniques  has 
led  to  a  demand  for  applications  in  a  very  different  environment: 
instead  of  depending  on  a  specially  trained  decision  analyst, 
come  decision  makers  would  prefer  to  acquire  a  computer-based 
decision  aid  which  could  operate  successfully  without  the  con¬ 
tinuous  need  for  a  professional  analyst. 
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Early  efforts  to  produce  decision  aids,  although  they  led 
to  valuable  research  findings,  did  not  succeed  as  field  appli¬ 
cations.  Even  when  the  targeted  users  agreed  that  the  computer- 
aided  decision  process  did  improve  decision  making,  these  aids 
eventually  fell  into  disuse  when  the  personal  support  of  the 
decision  analysts  was  removed.  Among  the  many  hypotheses  to 
explain  this  phenomenon,  the  following  appeared  to  provide  the 
most  powerful  explanation: 

In  the  absence  of  strong  external  motivation, 
the  general  user  will  prefer  a  computer-aided 
decision  process  only  if  it  is  perceived  as  a 
convenient  way  of  saving  time  and  effort. 

In  other  words,  user  attitudes  tend  to  be  dominated  by  the 
short-range  considerations  of  speed,  convenience,  and  simpli¬ 
city  of  operation,  rather  than  by  the  more  qualitative  factors 
like  methodological  thoroughness  and  consistency,  or  hard-to- 
see  impacts  such  as  a  tendency  to  make  better  choices  with  the 
decision  aid. 

DD I  undertook  this  project  to  demonstrate  and  test  this 
hypothesis.  The  purpose  is  to  (1)  produce  an  aid  which  would 
he  pleasurable  rather  than  painful  to  operate;  (2)  save  rather 
than  consume  time;  and  (3)  fit  into  the  user's  office  cr  field 
site,  while  still  providing  the  benefits  of  a  technically  sound 
decision-analytic  approach.  In  particular,  these  criteria  were 
specified : 

o  The  aid  should  be  small  enough  to  fit  comfortably 
into  an  office,  and  should  require  no  special  in¬ 
stallation  procedures. 

o  It  should  be  simple  to  operate,  and  not  require  users 
manuals  or  training. 


o  It  should  provide  self-contained  instruction  for  the 
novice  user,  while  permitting  the  more  experienced 
user  to  bypass  unnecessary  explanatory  material. 

o  User  inputs  should  be  simple,  rapid,  and  convenient; 
in  particular,  the  need  to  type  inputs  should  be 
minimized . 

o  As  far  as  possible,  user-machine  interactions  should 
avoid  methodology  and  specialized  technical  vocabu¬ 
lary. 

o  It  should  be  possible  for  the  user  to  request  help, 
to  terminate  a  session,  or  to  discontinue  a  session 
temporarily  and  resume  later,  at  any  point  during 
the  analysis. 

o  Results  should  be  presented  to  the  user  in  a  simple, 
visually  appealing  format,  using  graphics  wherever 
possible,  in  preference  to  text  or  numbers. 

o  A  library  of  prepared  analytic  models  should  be 

available  for  users  at  all  levels  of  experience  to 
apply  in  a  variety  of  problem  areas. 

o  It  should  be  possible  for  a  more  experienced  user  to 
construct  new  models  to  apply  to  arbitrary  ad  hoc 
problems . 

o  The  hardware  configuration  of  the  system  should  be 
producible  in  large  quantities  at  a  low  unit  cost 
($25,000  would  be  an  upper  bound). 


3 


1.2  Summary 


The  remainder  of  this  report  describes  DPI’s  efforts  to 
achieve  the  objectives  listed  in  Section  1.1.  It  also  describes 
the  degree  of  success  attained,  and  the  lessons  learned  from  the 
effort.  Section  2.0  describes  the  technical  approach  DDI  used, 
including  the  decision-analytic  methods,  user-engineering  ap¬ 
proach,  and  system  configuration.  In  Section  3.0,  the  decision 
aid  product  is  evaluated  with  respect  to  the  criteria  listed  in 
Section  1,1.  Section  4.0  summarizes  the  lessons  learned  during 
the  process  of  research  and  development.  Finally,  Section  5.0 
contains  recommendations  for  further  development  and  improvement, 
and  for  transfer  to  users  in  the  field. 


2.0  TECHNICAL  APPROACH 


The  decision-analytic  approach  of  the  Friendly  Decision 
Aid  is  described  in  detail  in  Section  2.1.  Briefly,  it  is  a 
variant  of  the  EVAL/HIVAL  approach  to  Multi-Attribute  Utility 
Analysis,  with  a  simplified  procedure  for  eliciting  the  quan¬ 
titative  input  values  and  for  displaying  results.  Decision 
templates  provide  the  user  with  a  library  of  previously  devel¬ 
oped  model  structures?  in  addition,  the  experienced  user  can 
also  use  the  aid  to  construct  new  models  for  ad  hoc  problems. 

While  the  decision-analytic  techniques  used  are  fairly 
standard,  the  efforts  at  user-friendly  interaction  represent  a 
new  and  unique  approach.  Section  2.2  describes  the  psycho¬ 
logical  basis  for  the  aid's  interactive  features,  while  Sec¬ 
tion  2.3  summarizes  the  system's  overall  configuration. 

2 . 1  Decision-Analytic  Approach 

2.1.1  Multi-Attribute  Utility  Analysis  -  The  EVAL/HIVAL 
methodology  was  selected  for  several  reasons.  It  involves 
scoring  each  of  the  alternative  options  on  several  fundamental 
criteria,  and  then  weighting  each  of  the  criteria.  For  simpli¬ 
city  and  transparency,  criteria  may  be  grouped  hierarchically 
into  superordinate  categories,  with  a  manageable  number  of  sub¬ 
ordinate  criteria  (two  to  four)  per  category.  Since  the  evalu¬ 
ation  model  is  a  linear  combination,  it  is  robust  mathematically, 
and  thus  is  ordinarily  not  sensitive  to  small  changes  in  the 
input  values.  In  addition,  the  associated  model  is  easy  to 
describe,  and  does  not  require  the  understanding  of  probability 
theory. 

The  elicitation  of  option  scores  in  a  HIVAL  model  is 
generally  a  straightforward  process.  Taking  each  criterion 
separately,  the  user  is  first  asked  to  select  the  best  and  the 
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worst  option  with  respect  to  a  criterion.  Then,  the  remaining 
options  are  rated  on  a  scale  between  the  two  extremes.  While 
most  previous  decision  aids  have  used  a  numerical  scale  (e.g., 

0  to  100  points)  for  this  elicitation,  the  Friendly  Decision 
Aid  displays  a  linear  scale  graphically  and  interacts  with 
the  user  in  terms  of  the  location  of  points  along  that  scale 
(retaining  the  numerical  equivalents  internally);  the  user 
never  needs  to  specify  inputs  numerically  or  to  interpret 
numerical  outputs. 

More  difficulty  is  typically  associated  with  the  assess¬ 
ment  of  weights  for  the  criteria.  This  step  involves  a  more 
complex  judgment:  the  decision  maker  is  asked  to  judge  the 
relative  importance  of  the  difference  between  the  best  and  the 
worst  options  on  one  criterion,  as  compared  with  the  difference 
between  best  and  worst  on  another  scale.  Because  it  requires 
the  decision  maker  to  consider  several  quantities  simultane¬ 
ously  while  making  critical  value  judgments,  this  step  is 
difficult  to  describe  simply.  For  this  reason,  and  because  of 
the  task's  unfamiliarity  and  inherent  complexity,  this  step  is 
the  most  frequent  source  of  major  errors  in  the  assessment 
process. 

The  solution  implemented  in  the  Friendly  Decision  Aid  is 
to  ask  the  user  for  a  series  of  simpler  judgments,  involving  a 
simple  ranking  of  criteria,  and  inferring  from  the  rank-ordering 
judgments  the  implied  set  of  criterion  weights.  This  general 
type  of  procedure  is  known  as  conjoint  measurement  and  has  a 
large  body  of  psychological  and  mathematical  research  behind  it. 
In  the  Friendly  Decision  Aid,  after  the  options  have  been  scored, 
the  aid  presents  those  criteria  on  which  a  given  option  ("Option 
A")  was  preferred  to  another  ("Option  B").  The  user  rank  orders 
those  criteria  according  to  the  magnitude  of  the  preference  for 
Option  A  over  Option  B,  From  a  series  of  such  rank  orderings, 
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the  aid  calculates  a  complete  set  of  weights  consistent  with 
the  user’s  judgments. 


2.1.2  Decision  Templates  -  Certain  decision  problems  can 
be  partially  anticipated.  For  instance,  although  the  specific 
options  and  their  values  may  not  be  known  in  advance,  the  eval 
uation  criteria  and  structure  can  be  developed  and  stored  for 
more  rapid  use  when  a  specific  decision  is  required.  These 
problems  fall  into  three  general  categories: 

o  Familiar  problems  which  may  occur  for  a  large  popu¬ 
lation  of  decision  makers  who  will  tend  to  use  the 
same  general  set  of  criteria  with  only  minor  varia¬ 
tions  (e.g.,  buying  a  car,  or  choosing  a  vacation 
site) . 


o  Recurring  problems,  where  the  same  user  routinely 

faces  the  same  type  of  decision  problem  often  enough 
to  adopt  a  standardized  approach  (e.g.,  preparation 
of  personnel  evaluations). 

o  Anticipated  crises,  where  the  critical  need  for  a 
rapid  response  places  a  premium  on  analysis  in 
advance  of  the  actual  crisis  (e.g.,  preparation  for 
an  enemy  attack) . 


In  such  situations,  it  is  advantageous  for  the  decision 
maker  to  have  access  to  a  previously  developed  model  structure 
which  can  be  completed  simply  by  specifying  options,  and  asses 
sing  scores  and  weights.  These  structures,  or  Decision  Tem¬ 
plates,  help  the  Friendly  Decision  Aid  to  minimize  the  impact 
of  model  structuring  which  can  be  the  most  time-consuming  and 
risky  phase  of  decision  analysis.  By  providing  a  library  of 
templates  for  general  users,  the  Friendly  Decision  Aid  permits 
even  a  novice  user  to  perform  a  limited  range  of  decision 
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analyses  without  difficulty.  Also,  by  permitting  the  more 
sophisticated  user  to  build  and  store  new  templates,  the  aid 
helps  the  decision  maker  to  allocate  time  more  effectively. 

2.1.3  Decision-analytic  expertise  -  In  conducting  an 
applied  decision  analysis,  the  professionally  trained  analyst 
does  not  adhere  to  a  rigid,  linear  sequence  of  steps.  The  ex¬ 
perienced  analyst  knows  when  to  examine  an  issue  in  more  detail 
and  when  a  rough  approximation  will  suffice;  he  recognizes  when 
and  how  to  apply  alternative  elicitation  techniques,  and  how  to 
minimize  possible  biases  in  judgments;  and  he  knows  when  and  how 
to  revise  the  structure  of  a  developing  model  as  new  information 
appears.  To  incorporate  such  procedural  knowledge  in  a  computer- 
based  aid  would  greatly  improve  its  value,  particularly  in  guid¬ 
ing  the  construction  of  new  models.  -Mhile  the  overall  approach 
of  the  Friendly  Decision  Aid  would  lend  itself  to  the  creation 
of  such  a  "smart"  aid,  only  the  simplest  of  approaches  have  been 
implemented  in  the  current  version. 

2 . 2  Psychological  Approach 

The  following  design  goals  are  based  on  research  findings 
in  behavioral,  cognitive,  and  human  factors  psychology.  Com¬ 
bined  with  the  simple  decision-analytic  approach  discussed  in 
the  preceding  section,  they  form  the  basis  for  DDI's  approach 
to  the  Friendly  Aid. 

o  Easy  access  -  Because  the  most  immediate  consequen¬ 
ces  have  the  strongest  influence  on  behavior,  the 
user  should  have  no  difficulty  in  starting  up  the 
system,  and  no  prior  training  should  be  necessary. 

o  Simple  controls  -  To  minimize  user  error  and  effort, 
controls  sho-?d  be  simple  and  direct;  in  particular, 
the  need  for  entering  text  via  a  typewriter  keyboard 
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should  be  minimized.  Also,  wherever  possible,  the 
user  should  be  presented  with  a  limited  set  of 
well-defined  choices,  rather  than  required  to  know 
what  options  are  available. 

o  Sensorimotor  feedback  -  To  maintain  a  reliable  link 
for  user-machine  communication,  user  inputs  should 
be  immediately  acknowledged  by  a  perceptible  (but 
non-disruptive)  signal  from  the  aid;  in  the  event  of 
an  "illegal"  user  response,  the  aid  should  let  the 
user  know  quickly,  allowing  the  user  to  correct  the 
response  without  disrupting  the  aid's  ability  to 
function . 

o  Response  timing  -  Ideally,  the  aid  should  perform  all 
of  its  functions  quickly  so  the  user  never  has  to  wait 
more  than  a  few  seconds  for  a  response.  If  a  delay  of 
more  than  30  seconds  is  inevitable,  however,  the  aid 
should  warn  the  user  and,  if  possible,  indicate  how 
long  a  delay  to  expect. 

o  Adaptation  to  individual  users  -  To  make  the  aid  more 
compatible  with  each  user's  personal  preferences  and 
behavior  patterns,  the  aid  should  "customize"  its 
functions:  fast/slow  tracking  should  permit  the  more 

experienced  user  to  bypass  things  like  tutorial  mater¬ 
ial,  which  the  novice  would  find  useful;  user  prefer¬ 
ences  for  modes  of  input  and  output  should  be  accommo¬ 
dated  as  well. 

o  Aesthetics  -  The  aid  should  be  aesthetically  attrac¬ 
tive  to  the  user.  It  should  be  packaged  in  a  secure, 
sturdy,  neat-looking  cabinet  which  can  fit  unobtru¬ 
sively  into  an  office. 
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2 . 3  System  Configuration 


The  actual  configuration  of  the  DDI  Friendly  Decision  Aid 
consists  of  an  Apple  II  personal  computer,  augmented  by  a  10  Mb 
hard  disk.  All  input  and  output  functions  take  place  on  a  color 
video  monitor  with  a  superimposed  transparent  touch-sensitive 
screen.  Because  the  user  interacts  only  with  this  touch-screen 
display,  the  other  components  have  been  enclosed  in  a  cabinet 
which  not  only  simplifies  the  system's  appearance,  but  also 
enhances  portability  and  reduces  the  risk  of  accidental  damage. 
The  system  is  arranged  so  that  the  monitor's  on/off  switch  auto¬ 
matically  sequences  all  hardware  components  and  triggers  sign-on 
and  sign-off  routines;  no  "log-in”  or  "log-off"  procedures  need 
to  be  learned,  and  there  is  no  need  in  normal  operation  to  see 
or  touch  any  of  the  components  within  the  cabinet. 

While  the  aid  is  functioning,  the  user  enters  information 
by  touching  areas  on  the  screen.  A  number  of  formats  for  this 
interaction  are  used  during  the  aid's  operation.  For  example, 
control  functions  are  selected  from  push-button-like  squares 
at  the  bottom  of  the  display.  Selection  of  items  from  a  list 
(e.g.,  in  response  to  a  question  such  as,  "which  of  these  options 
is  best  on  this  factor?")  takes  place  when  the  user  touches  the 
chosen  item  on  the  screen.  (A  short  immediate  "beep"  alerts 
the  user  that  his  response  has  been  accepted;  if  the  user  has 
touched  an  "illegal"  section  of  the  screen,  a  longer,  lower- 
pitched  sound  warns  the  user  to  reenter  the  response.)  Although 
the  use  of  restricted-choice  menus  avoids  most  of  the  need  for 
free  text  input,  the  display  of  a  typewriter  keyboard  (with  the 
characters  either  in  alphabetical  order  or  in  the  normal  type¬ 
writer  configuration)  is  available  on  which  the  user  can  "type" 
by  touching  the  areas  on  the  screen  corresponding  to  the  desired 
"keys" . 
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The  aid’s  software  permits  the  user  to  identify  himself 
and  to  specify  certain  personal  preferences  which  are  then 
maintained  in  a  user  profile.  For  example,  the  user  may  re¬ 
cord  a  preference  for  the  alphabetical  keyboard  display  versus 
the  typewriter  configuration.  In  addition,  the  user  profile 
includes  information  about  the  user's  level  of  experience  on 
the  system  ("novice"  versus  "fast  track"),  and  a  library  of 
templates  and  user-constructed  models. 
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3.0  PRODUCT  EVALUATION 


The  DDI  Friendly  Decision  Aid,  as  realized  in  the  proto¬ 
type  version  constructed  under  the  present  contract,  demon¬ 
strates  several  positive  features,  some  for  the  first  time  on 
any  functional  decision-aiding  system.  In  general,  the  re¬ 
sponses  of  pilot  users  have  been  positive,  although  no  formal 
user  evaluation  has  been  attempted  to  date.  A  few  negative 
observations  have  been  noted,  however,  which  are  related  to 
the  specific  hardware  chosen  for  the  prototype  system  (partic¬ 
ularly  the  use  of  the  Apple  II  computer).  Furthermore,  some 
absent  capabilities  were  identified  as  desirable  additions  to 
any  subsequent  version.  Finally,  a  number  of  features  which 
were  part  of  the  original  plan  could  not  be  sufficiently  devel¬ 
oped  during  the  span  of  the  contract  to  permit  implementation 
and  testing. 

3 . 1  Good  Features 

3.1.1  Analytic  approach  -  3y  restricting  the  judgments 
required  of  the  user  to  simple  ordinal  rankings,  and  by  avoid¬ 
ing  the  need  for  numerical  assessments  entirely,  the  Friendly 
Decision  Aid  overcomes  perhaps  the  most  significant  objection 
of  untrained  decision  makers:  their  perceived  inability  to 
quantify  subjective  judgments.  In  addition,  because  results 
are  depicted  graphically  rather  than  numerically,  the  effort, 
delay,  and  risk  of  error  associated  with  interpreting  those 
results  is  substantially  reduced. 

3.1.2  User-machine  interaction  -  The  simplified  inter¬ 
action  using  the  touch-screen  display  for  all  inputs  and 
outputs  was  very  successful.  It  reduced  response  effort  to  a 
minimum,  while  permitting  the  user  to  focus  attention  on  a 
single  control  area  (rather  than  shifting  attention  from  a 
display  screen  to  a  keyboard,  a  printer,  a  diskette  drive. 
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etc.).  The  immediate  audio  feedback  ("beep")  accompanying 
each  touch-screen  entry  helped  to  maintain  user  confidence 
and,  consequently,  to  permit  faster  operation  with  less  need 
for  checking  entries. 

The  use  of  color,  while  not  absolutely  essential  to 
the  aid's  functioning,  was  extremely  valuable  from  a  motiva¬ 
tional  and  from  a  human  factors  standpoint.  The  multi-colored 
displays  added  visual  stimulation  and  variety  to  the  user  envi¬ 
ronment.  Furthermore ,  it  made  the  graphical  displays  easy  to 
read,  and  permitted  the  use  of  color-coding  to  identify  groups 
of  items  belonging  to  the  same  category. 

Another  display-related  feature  which  served  both 
motivational  and  human-factors  goals  was  the  use  of  simple 
graphic  displays  on  which  changes  could  be  represented  as 
apparent  motion.  For  example,  a  change  in  one  or  more  of  the 
options'  scores  would  appear  on  the  display  as  the  movemer t  of 
the  corresponding  colored  dots.  If  a  weight  was  changed,  the 
implied  change  in  overall  scores  would  appear  as  the  growth  or 
shrinkage  of  the  bars  which  represent  chose  scores.  This 
motion,  especially  with  a  rapid  response,  made  the  aid  more 
interesting  to  operate,  while  making  the  results  of  any  changes 
easy  to  visualize. 

3.2  Bad  Features 


Most  of  the  undesirable  features  associated  with  the 
Friendly  Decision  Aid  could  be  traced  to  the  use  of  the  Apple  II 
personal  computer  as  a  central  processing  unit.  Originally 
purchased  because  of  its  low  basic  cost  and  wide-spread  availa¬ 
bility,  the  Apple  II  appeared  to  be  the  best  of  the  "personal" 
microcomputers  available  commercially  at  the  onset  of  the 
contract.  However,  the  low  price  turned  out  to  be  somewhat 
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deceptive,  as  peripheral  devices  had  to  be  added  to  augment 
the  basic  system's  capabilities  to  supporc  the  aid's  functions. 

The  Apple  II  processor  itself  was  designed  as  a  hobbyist's 
computer,  and  lacks  the  size,  speed,  flexibility,  and  features 
that  a  professional  application  would  require.  Its  memory 
limitations  impose  constraints  both  on  the  system's  capability 
and  on  the  design  and  programming  processes.  Its  operation 
using  the  Pascal  language  was  often  so  slow  that  important 
routines  had  to  be  rewritten  when  a  timely  result  rather  than 
a  slower,  but  more  correct  one  was  required;  options  for  faster 
languages  were  limited,  and  on  one  routine,  even  a  recoding  in 
assembly  language  failed  to  improve  time  performance.  Finally, 
the  Apple  II  graphics  (and  particularly  the  alphanumeric  char¬ 
acter  set)  were  unwieldy,  producing  irregular  coloration  and 
poor  legibility;  this  necessitated  dditional  efforts  to  cor¬ 
rect  the  resulting  display  problems  (e.g.,  the  construction  of 
a  new,  more  legible,  character  set). 

A  related  problem  which  was  an  outgrowth  of  the  hardware 
configuration  was  the  excessive  bulk  of  the  entire  system.  A 
more  compact,  integrated  unit  in  a  smaller  cabinet  would  have 
been  possible,  given  a  different  choice  of  hardware. 

In  the  decision  aid  itself,  the  only  major  shortcoming 
was  the  lack  of  a  "road  map"  to  answer  the  user's  "where  am  I?" 
questions.  Occasionally,  during  the  course  of  an  analysis  or 
upon  resuming  a  previously  interrupted  analysis,  the  user  needs 
to  be  reminded  of  his  current  stage  and  how  much  remains  to  be 
done.  Alternatively,  the  user  may  be  focusing  on  one  portion 
of  an  analytic  model,  and  wish  to  see  where  it  appears  in  the 
context  of  the  entire  model.  Demands  for  a  display  of  the  en¬ 
tire  model  or  the  entire  procedural  context,  without  disrupting 
the  flow  of  the  analysis,  are  currently  unmet.  Other  than  in¬ 
herent  limitations  on  the  size  of  the  display,  there  should 
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be  no  obstacle  to  adding  an  optional  "where  am  I"  feature  to 
the  current  system. 

3.3  Untested  Features 


Three  elements  of  the  originally  planned  design  were  not 
sufficiently  developed  during  the  research  effort  to  permit 
their  implementation,  testing,  and  evaluation.  The  first  — 
voice  input  and  output  —  was  begun  using  a  speech  recognizer 
for  input  of  keywords  (with  a  limited  vocabulary  of  words)  and 
a  speech  synthesizer  for  output  of  verbal  material.  Although 
some  initial  success  was  achieved,  the  memory  requirements  pre¬ 
cluded  its  use  in  the  final  version.  Ultimately,  more  effort 
will  be  needed  to  incorporate  a  reliable  speech  input/output 
facility  into  the  decision  aid. 

A  second  feature  which  was  not  implemented  during  this 
initial  effort  was  an  instruction  capability,  in  which  the  aid 
would  explain  every  step  of  the  analysis  to  the  naive  user  by 
displaying  an  exemplary  session  with  a  hypothetical  user. 

Partly  because  the  aid  functions  so  simply,  and  partly  because 
a  "help"  feature  would  provide  needed  information  on  demand, 
the  instructional  capability  was  not  a  high-priority  item. 

A  more  desirable  feature,  but  one  which  might  require  a 
substantial  effort  to  implement  satisfactorily,  would  be  to 
add  sufficient  "intelligence"  to  guide  the  analytic  process. 
Incorporating  some  of  the  procedural  practices  and  heuristics 
of  the  experienced  decision  analyst,  such  a  routine  could 
streamline  the  analysis,  and  at  the  same  time,  produce  more 
compelling  results.  At  present,  no  existing  system  demonstrates 
a  successful  capability  of  this  sort,  so  the  potential  benefit 
of  a  success  would  be  great. 
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4.0  LESSONS  LEARNED 


This  section  presents  a  brief  summary  of  several  lessons 
learned  during  the  course  of  this  research  effort.  Some  of 
those  lessons  refer  to  the  functional  properties  of  decision 
aid  systems,  some  to  specific  experiences  with  respect  to  the 
selection  and  purchase  of  hardware  components,  and  some  to  the 
design  process  itself. 

4 . 1  Functional  Properties 

The  ma-ior  lesson  learned  from  this  effort  was  that  a  sim¬ 
ple,  user-friendly  system  on  a  small  computer  is  a  feasible 
undertaking.  The  se  of  simplified  input/output  via  the  touch¬ 
screen  display  was  very  successful,  as  was  the  use  of  color  and 
simple  auditory  feedback  to  maintain  user  attention  and  motiva¬ 
tion.  V7hile  the  use  of  speech  to  augment  the  visual  displays 
was  not  fully  tested,  it  appears  to  be  a  desirable  direction 
for  continued  research. 

In  terns  of  the  analysis  itself,  the  use  of  templates  to 
provide  a  general-purpose  library  for  the  user  made  it  possible 
for  even  the  novice  user  to  perform  a  complete  analysis  fairly 
quickly.  The  non-numer ical  assessment  and  display  techniques 
(ordinal  judgments,  graphic  displays,  and  motion  to  represent 
change)  worked  successfully,  and  deserve  more  widespread 
application  in  decision  aids. 

Although  the  current  effort  did  not  attempt  to  implement 
a  dialogue  system  in  which  the  aid  explains  both  the  decision- 
analytic  theory  and  specific  procedures  to  the  user,  experience 
with  the  current  system  suggests  that  such  a  system  might  not 
be  necessary,  and  indeed  that  extensive  dialogue  might  even  be 
undesirable.  In  the  first  place,  a  similar  informative  capabil¬ 
ity  could  be  achieved  by  extensive  use  of  the  "help"  button 


feature  which  would  make  supplementary  information  available 
without  any  penalty  to  the  user  who  does  not  need  that  informa¬ 
tion,  or  who  cannot  spare  the  time.  Furthermore,  the  success 
of  the  simplified  analytic  ^dures  in  the  current  aid  makes 

the  need  for  such  explanatory  material  far  less  critical  than 
it  would  be  if  more  complex,  sensitive  judgments  were  desired. 

4 . 2  Hardware  Experience 

The  primary  lesson  learned  from  the  present  research  and 
development  experience  is  that  a  professional  quality  decision- 
aiding  system  requires  more  sophistication  than  the  Apple  II  or 
a  similar  home  computer  can  ordinarily  provide.  The  option  of 
adding  on  peripheral  devices  to  augment  its  capability  is  ex¬ 
pensive  enough  to  negate  the  original  cost  and  size  advantages, 
and  even  with  substantial  additional  effort,  cannot  match  the 
capabilities  of  some  larger,  only  slightly  more  expensive  micro 
computers.  In  a  market  where  hardware  costs  in  general  are 
dropping,  and  labor  and  software  costs  arc  rising,  a  more  capa¬ 
ble  processor  should  be  worth  the  additional  initial  investment 

Considerations  which  should  influence  the  choice  of  a 
central  processor  include  physical  size,  memory  capacity, 
peripheral  (disk)  speed  and  capacity,  and  overall  reliability 
and  service.  In  addition,  the  following  factors  are  critical: 

o  Good-quality  graphics,  including  an  easily  legible 
type  font; 

o  Input/output  compatibility  with  color  video  monitors 
(RGB  and  STSC) ,  as  well  as  large-screen  video  projec¬ 
tors;  and 

o  Ability  to  perform  calculations  rapidly,  without 
disrupting  display  or  control  functions. 
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Another  hardware-related  lesson  learned  during  this 
effort  was  that,  at  least  at  present,  microprocessor  hardware 
delivery  is  routinely  subject  to  delays  of  up  to  about  four 
weeks.  In  addition,  personal  computer  vendors  routinely  offer 
equipment  which  is  not  yet  available,  in  an  attempt  to  gauge 
market  response.  The  limitations  and  idiosyncrasies  of  much 
microprocessor  hardware  can  also  cause  further  delays  due  to 
programming  difficulties  because  non-standard  techniques  must 
be  used  to  save  space  01  time.  This  programming  delay  is  com¬ 
pounded  by  the  limited  selection  of  high-level  programming 
languages  on  some  microcomputers. 

4 . 3  The  Design  Process 


The  experience  of  this  and  previous  efforts  at  designing, 
implementing  and  evaluating  computer-based  decision  aids 
suggests  that  the  design  of  a  complex  interactive  system  such 
as  a  decision  aid  should  be  an  evolutionary  process.  The 
"linear"  approach  in  which  SDecif ications  are  determined  by 
the  analyst  and  are  passed  on  to  computer  scientists  who  try 
to  meet  those  specifications  is  not  only  inefficient,  but  it 
also  runs  the  risk  of  producing  a  useless  product.  This  may 
occur  because  minor  flaws  in  the  specifications  or  their  com¬ 
munication  may  drive  the  development  in  the  wrong  direction. 

The  evolutionary  approach,  on  the  other  hand,  involves  a  less 
structured,  iterative  process  in  which  there  is  ample  oppor¬ 
tunity  for  the  analyst/designer  or  the  target  user  to  inter¬ 
vene  before  significant  resov'es  are  committed.  Ideally,  the 
system  should  be  as  flexible  as  possible  during  the  development 
process,  even  if  this  flexibility  involves  moderate  sacrifices 
in  cost,  speed,  or  size;  only  after  a  final  design  has  been 
implemented,  tested,  and  accepted,  should  efforts  be  specifi¬ 
cally  directed  to  improving  efficiency. 
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For  a  decision  aid  such  as  the  Friendly  Decision  Aid,  the 
implications  of  this  design  philosophy  would  suggest  that,  in 
the  future,  similar  aids  be  initially  developed  on  a  large, 
fast,  flexible  system,  using  a  high-level,  easily  modifiable 
language  (such  as  APL) .  On  such  a  system,  the  designers  would 
be  able  to  attend  to  effectiveness,  without  the  unnecessary 
inconvenience  and  delays  involved  in  conforming  to  a  more  re¬ 
strictive  environment.  Iterative  testing  and  improvement,  or 
maior  redesigns,  could  take  place  in  parallel  on  a  modular 
basis,  with  the  option  of  assigning  different  "experts"  to 
different  subtasks. 

With  proper  precautions,  the  final  form  of  the  aid  could 
be  transferred  satisfactorily  to  the  actual  system  on  which 
the  aid  will  operate.  Even  if  this  transfer  involves  recoding 
in  a  different  language  or  adaptations  for  different  hardware, 
the  total  time  required  for  the  development  effort,  the  man¬ 
power  and  development  cost,  and  the  quality  of  the  final  pro¬ 
duct  should  all  improve.  The  evolutionary  process  as  described 
here  allows  productive  development  to  take  place  according  to 
plan,  even  if  delays  in  delivery  or  service  problems  make  the 
system  hardware  temporarily  unavailable.  Because  of  increased 
flexibility  in  making  hardware  changes  (e.g.,  in  response  to 
new  information  or  new  products) ,  risk  of  premature  commitment 
to  a  particular  item  or  vendor  is  further  reduced. 
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5.0  RECOMMENDATIONS 


The  research  efforts  described  in  this  report  have  resulted 
in  a  convincing  demonstration  that  a  computer-based  decision 
aid  can  be  designed  for  effective  use  in  the  absence  of  a  deci¬ 
sion  analyst.  The  prototype  system  which  this  report  accom¬ 
panies  represents  a  major  advance  in  terms  of  the  criteria  for 
a  successful  decision  aid.  However,  there  are  many  directions 
for  continued  growth  and  development.  This  section  summarizes 
a  few  recommended  directions  for  future  research  and  development. 

5 . 1  Hardware  Improvements 

During  its  initial  investigations,  DDI  detected  a  number 
of  features  in  the  hardware  design  of  the  Apple  II  computer 
which  severely  limit  the  capabilities  of  Apple -b<;sed  decision 
aids.  Particularly  significant  are  the  limits  on  color  graphic 
display  and  the  lack  of  input/output  buffering.  Because  of 
the  way  Apple  implements  its  color  graphics,  only  six  colors 
are  available,  and  these  cannot  be  mixed  freely  on  the  screen. 

In  particular,  true  colored  alphanumeric  characters  are  not 
possible,  and  ordinary  white  characters  are  displayed  with 
random  tints  of  colors.  Because  displays  are  limited  to  only 
40  characters  per  line,  it  is  often  necessary  to  abbreviate 
labels  (for  options  and  criteria),  an  undesirable  feature  from 
the  user's  point  of  view.  Furthermore,  the  color  signal  from 
the  Apple  contains  extraneous  noise  which  causes  graphics  to 
be  blurred.  Finally,  the  lack  of  input/output  buffering  means 
that  the  execution  of  the  decision-analytic  routines  must  halt 
while  program  segments  or  audio  output  phrase  tables  are  being 
loaded  into  computer  memory. 

While  it  would  be  premature  to  specify  a  unique  configura¬ 
tion  for  replacement  hardware,  a  number  of  attractive  alterna¬ 
tives  have  been  identified  and  approximate  costs  assessed. 
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With  the  intended  changes  in  hardware  (consisting  primarily  of 
a  different  central  processor  and  graphics  display),  the  new 
prototype  systems  should  cost  approximately  525,000  per  unit. 
However,  S5000  can  be  saved  by  reusing  components  of  the  ori¬ 
ginal  systems,  with  substantial  cost  reductions  when  produced 
in  quantity. 

The  transfer  of  the  current  decision-aid  software  to  the 
new  configuration  will  necessarily  involve  a  moderate  amount 
of  technical  labor  in  addition  to  the  hardware  itself.  How¬ 
ever,  if  the  new  system  is  implemented  using  the  same  higher- 
order  language  (Pascal),  labor  costs  should  be  minimized. 


5 . 2  Full-Scale  Instruction  Capability 

As  originally  conceived,  the  Friendly  Decision  Aid  was 
intended  to  provide  an  instruction  capability  so  that  any 
decision  maker,  even  one  who  had  never  heard  of  decision 
analysis  before,  could  use  and  understand  the  aid.  In  order 
to  provide  such  a  capability  without  unnecessarily  slowing 
down  or  boring  the  more  experienced  user,  a  "slow/fast  track” 
feature  was  designed  into  the  system.  A  slow-track  user  would 
receive  more  explicit  directions  and  assistance;  the  fast 
track,  on  the  other  hand,  uses  abbreviations  and  interactive 
routines  to  permit  faster  and  more  convenient  operation. 

While  the  original  decision  aid  provides  some  instruction 
and  some  assistance  via  a  "help"  option,  it  currently  permits 
the  novice  user  to  construct  decision  models  only  by  using  the 
pre-developed  templates  in  its  memory;  only  fast-track  users 
can  create  completely  new  templates.  One  of  the  goals  of  the 
proposed  project  will  be  to  extend  this  template-building 
capability  to  the  slow  track,  while  providing  more  extensive 
and  more  useful  instruction  and  guidance. 


5.3  "Smart"  Decision  Aid 


The  current  Friendly  Decision  Aid  is  helpful  in  guiding 
the  decision  maker  through  a  standard  sequence  of  steps  in 
the  decision  analysis  process.  Although  the  overall  process 
is  highly  responsive  to  the  user's  directions,  it  lacks  the 
intuitive  ability  to  perceive  patterns  in  the  user's  responses 
which  might  indicate  the  need  fur  alternative  elicitation 
techniques,  or  the  possibility  of  time-saving  simplifications. 

A  further  advantage  could  be  achieved  from  the  capability  to 
help  the  user  explore  the  decision-analytic  model  in  an  attempt 
to  detect  underlying  patterns  in  the  data;  this  would  aid  the 
decision  maker's  understanding  of  the  problem,  and  help  to 
identify  and  reconcile  possible  inconsistencies. 

The  task  of  implementing  a  "smart  aid"  feature  should  be 
extremely  challenging,  as  it  requires  the  most  sensitive  and 
adaptive  aspects  of  the  decision  analyst's  relation  to  the 
decision  maker.  The  effort  will  involve  several  components: 
descriptive  behavioral  and  perceptual  studies  of  the  model 
exploration  process;  design  nd  implementation  of  a  more 
sophisticated  user-machine  dialogue  process;  and  the  develop¬ 
ment  and  coding  of  algorithms  to  be  used  as  heuristic  guides 
to  model  exploration. 

Although  the  task  is  ambitious,  the  potential  benefits  of 
a  "smart"  decision  aid  would  be  great.  Under  severe  time  con¬ 
straints,  the  user  could  advise  the  aid  to  look  for  shortcuts 
without  seriously  compromising  technical  validity.  On  the  other 
hand,  the  user  with  more  time  could  use  the  model  exploration 
feature  to  improve  the  model's  validity,  credibility,  and  use¬ 
fulness,  and  to  develop  a  more  compelling  summary  of  the  data 
in  the  model. 
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5 . 4  Formal  Testing  and  Evaluation 


In  addition  to  the  pilot  testing  which  is  a  necessary 
part  of  the  development  effort,  it  would  be  valuable  to  sub¬ 
mit  the  Friendly  Decision  Aid  to  a  more  formal  testing  and 
evaluation  study.  A  structured  set  of  pre-determined  per¬ 
formance  criteri  i  could  be  constructed,  including  objective 
measures  such  as  system  response  times,  behavioral  observa¬ 
tions  such  as  user  error  rates,  and  subiective  evaluations 
such  as  user  confidence  ratings.  Using  these  criteria  as  an 
evaluation  structure,  the  aid  could  he  tested  using  realistic 
problems  for  which  a  "correct"  solution  could  be  determined 
by  experts,  and  a  population  of  subiects  similar  to  the  class 
of  decision  makers  for  whom  the  aid  is  intended.  The  aid  could 
be  evaluated  with  respect  to  "absolute"  criteria  (such  as  user 
expectations) ,  or  in  comparison  to  unaided  approaches  or  to 
other  decisions  aids. 

One  clear  value  of  a  formal  test  and  evaluation  study 
would  be  the  ability  to  document  the  aid's  performance  for  a 
large  audience  of  potential  users,  through  journal  publications 
and  other  public  media.  In  addition  to  its  potential  contribu¬ 
tion  to  the  technology  transfer  process,  a  formal  study  would 
also  contribute  to  further  development  and  refinement  by  gen¬ 
erating  an  extensive  list  of  comments  and  suggestions,  and 
would  produce  an  indication  of  the  target  users'  priorities. 
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